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Introduction 

The application of skills and knowledge to a “real world” context entails a greater set of 
competencies than just technical proficiency, for both the student who sits the problem and 
the teacher who sets it. These competencies require unpacking. A greater understanding of 
these competencies would enable teachers to create tasks that test what they are supposed 
to, with improved insight into difficulty level. Moreover, it would enable students to receive 
not only a better training for those tasks, but also a more meaningful education. The 
reflections below are an attempt to begin this unpacking process and are based on my 
own teaching experiences. 

The current Year 12 format for problem solving assessment in Victorian education is 
'extended response', which is the model that I have been adopting in mathematics classes 
down to Year 9. (My current and former schools are 9-12 campuses.) In this context, 
students have access to their Computer Algebra System (CAS) calculators, but may be asked 
to prove certain results along the way using their by-hand skills. When one result is required 
in subsequent parts, it is standard to provide the answer in the question and ask students to 
“show that” result using appropriate algebraic techniques. 

The feedback I have recorded indicates that even technically strong students find these 
questions difficult. This is something of a mystery since the actual mathematics involved is 
usually not complex, indeed with CAS calculator technology, it often comes down to listing 
just the key equations or steps. This was particularly evident in my first year at Nossal High, 
a new select-entry government high school in the south-east of Melbourne, Victoria. 

Consider the extended response problem (Figure 1) that was set for a Year 9 mid-year 
examination. The units of work we had covered included linear algebra, equations and 
relations. The question was designed to test their understanding broadly across these 
different aspects of intermediate linear mathematics. 

y 



Figure 1. Year 9 mid-year examination extended response problem. 
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On Athletics Day, Josiah dropped a long pole onto the top corner of a 
rectangular platform. See the diagram on the previous page for a side view 
of the situation. Josiah was standing at the point (0,0) at the time. 

All units are in metres. 

a) Show that the height of the rectangular platform is 4a + 3, given 
that the pole has the equation y = 4x - 1 in the above diagram. 

b) Show that the perimeter of the rectangular cross-section featured 
above is 6 a + 12 . 

c) Determine how far Josiah is from the foot of the pole on the 
ground; assume Josiah’s feet end at the point (0, 0). 

d) If the perimeter determined in b), plus the distance along the 
ground from the pole to the platform equals 12.05 m, show that 
the value of a is -0.1. Full marks will be awarded for a correctly 
modelled equation leading to a solution. 

e) The pole reaches 4.8 m above the ground. Using the gradient of 
the pole, find the length of platform directly underneath it. 

Students at my school performed very poorly on this question, despite their high mean 
ability. Indeed, many of the students who showed they could appropriately substitute into 
formulas, accurately simplify expressions and solve linear equations, in earlier parts of the 
paper, had no idea how to respond to any part of this question. I had clearly underestimated 
its difficulty. I thought about it purely on the required technical skills for a solution, rather 
than the thinking skills required to draw on these techniques, or indeed even the clarity 
of the question itself. It begs the following questions about middle and upper secondary 
mathematics education. 

1. How can our teaching and learning practices develop better thinkers rather than 
just proficient technicians? In other words, how can we help our students find 
engagement with mathematical concepts rather than just solution procedures? 

2. How can we not only accurately gauge task difficulty, in extended response, 
applied contexts, when assessing aspects of the mathematics curriculum, but 
also test what we intend to? 


Conceptual versus procedural learning 

After our Year 9 students sat the above-mentioned examination, several students complained 
to me that the Josiah question (featured in Figure 1) seemed foreign to them. Some went so 
far as to claim “You didn’t teach us that!” (I’m sure many readers have heard that one before!) 

I believe this extreme reaction lies with the view that mathematics is about procedures 
rather than concepts. It seems to be borne out of student expectation that a mathematics test 
should canvass a range of routine problems that they have already been exposed to. (I would 
argue that there is certainly a place for that, but tests based exclusively on this strategy give 
a limited insight into a student’s true understanding.) It would seem that in their previous 
experience (largely prior to Nossal), all tests feature ‘familiar’ problems, and testing for them 
is about recognising the pattern for which routine belongs where. Under this model of assess¬ 
ment, question difficulty is judged on the complexity of the routine that needs to be applied, 
rather than the depth of thinking required for accessing a relevant solution strategy. 

At Nossal High, a great deal of care is spent developing mathematical concepts from prior 
understandings. However, in practice, only a portion of students engage with this introduc¬ 
tory part of our presentations. Many students appear to only 'tune in' when worked examples 
are given and processes have been laid out. For such students, building understanding at 
the conceptual level does not appear to equate to ‘success in mathematics’. To highlight this 
point, a Year 9 student asked me for help one day during the lesson. After a lengthy expla¬ 
nation covering the underpinning principles behind a solution, the student turns to me and 
says, “So when I see this, I do that?”. 
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At Nossal High, academic success is highly valued by the student body. However, this 
can sometimes come at a cost to real learning, where flexibility, creativity and insight play, 
too frequently, second fiddle to ‘correctness’. A cultural shift needs to occur where thinking 
is valued in its own right and engagement with ‘rich’, cross-conceptual problems are 
made routine. 

An interesting approach to incorporating original thinking into mathematics teaching, and 
giving students the opportunity to create approaches for themselves, was outlined by Sullivan 
and colleagues (2015). Their lesson structure included posing problems that connect different 
aspects of mathematics together and for which students have not been shown a solution, 
and giving all students opportunity to engage with the key ideas. Of course, these issues 
mentioned above are not restricted to intermediate mathematics, but rather have a flow on 
effect into their Year 11 and 12 studies, where curriculum constraints become tight. 


Rating difficulty levels for extended response tasks 

Does a problem solving ‘continuum’ exist, at least in the context of Victorian upper secondary 
school education? That is, if we unpack all the thinking skills involved in a given question at 
a particular level, can we determine its difficulty level via a problem-solving scale? Does such 
a continuum change from topic to topic, be it how we progress along it, or the elements that 
constitute it? What about problem solving across several topic areas, like on exams where 
one question may blend together several different units of work? How does this add 
to problem solving complexity, if at all? 

A lot of valuable work in the area of rating ‘item demand’ has occurred through the 
Programme for International Student Assessment (PISA), which, since 2000, has been testing 
mathematical ability in 15 year-olds across the Organisation for Economic Co-operation and 
Development (OECD) countries. As a means for ‘objectively’ assessing question difficulty, 

PISA has developed competency frameworks over six different areas of problem solving. 
(Interestingly, only one of the six entails a student’s ability to work with symbols, operations 
and formal conventions.) Considerable effort has gone into defining and re-defining these six 
different areas of the framework so that cross-over is avoided and item ‘raters’ become more 
uniform in their assessment of item demand. The insights gained from PISA should help in 
the above mentioned project. The point of difference, however, is that, when contending with 
students of such diverse educational background, particularly in terms of their curricula, 
PISA focusses only on cognitively oriented definitions. By contrast, as a Victorian mathe¬ 
matics educator, it is important to know how previous experience affects students’ ability 
to problem-solve within areas of the curriculum, as well as across them and beyond them. 
Nevertheless, it provides a starting point for the study. 

Re-consider the Year 9 mid-year examination extended response problem presented 
in Figure 1. Part (a) required students to think about the ‘height of the platform’ in terms 
of the Cartesian plane on which the diagram was set. This thinking skill, captured broadly 
by PISA as ‘representation’, appeared to be beyond the capacity of many students. Should 
this lack of skill be attributed to their limited experience with problem solving within the 
Cartesian plane or is there something developmentally challenging about this kind of thinking 
for the average 15 year old? Perhaps in my years of experience I overlooked the conceptual 
gap between Josiah’s tip toes being at a certain location in physical space and the mathemat¬ 
ical idealisation of a singular point, (0, 0), at which they are purported to occur? That is, the 
problem is a mix of ‘reality’ and ‘mathematical reality’ and perhaps students are getting lost 
transitioning between the two? Furthermore, even if students could transition between the 
physical elements of the question and the Cartesian diagram, they still needed to realise that 
the height of the platform was equal to the y-value of the pole at the point of contact. And 
this leads to a further thinking skill that PISA labels, 'devising a strategy’. 
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The skill of finding the y-value of a relation, for given values of x, was well drilled. 

However, finding y when x is ‘a + T proved the stumbling block for some at this ‘strategy 
stage’, even though applying the distributive law (expanding brackets) and then collecting 
like terms was also well practised. This shortcoming I am suggesting lies with the student’s 
understanding, but perhaps it lay with me in the posing of the question itself. Could the 
“show that” element of the question be the source of confusion rather than the thinking skills 
required for an answer? (Some students in higher year levels, with greater exposure to these 
“show that” questions, continue to struggle with this style of question. It has motivated the 
study of mathematical proof in our Year 11 Specialist Mathematics course.) Should I have 
simply asked students to “Find the height of the pole above the ground, in terms of a”? Would 
this have made the question more accessible? 

The same concern relates to part (b), but as part (d) required the answer to part b), the use 
of “show that” seems more imperative. (The 2013 Mathematical Methods Exam 2 contained 
an extended response question with a part that really should have been expressed as a 
“show that”. Many students were unable to solve this part and thus were denied access to 
the remaining 6 marks of the question. On questioning the examiner about this, his reluc¬ 
tance came from the point of view of assessment. It was felt that if it was expressed as a 
“show that”, ‘fudged’ answers may have been hard to distinguish from genuinely derived 
ones. No such issues arose on the 2014 paper.) 

I was genuinely surprised to find widespread struggling by students with part (c). The 
majority of students who sat the paper were able to determine the x -intercept of a linear 
function such as this one, but of those, many had no idea that it directly related to this part 
of the question. It would seem that the transition between ‘reality’ and ‘mathematical reality’ 
again proved their undoing. 

With regards to part (d), students were required to subtract their part (c) answer from 
‘a + I’, add it to the expression for perimeter provided in part (b), and set this equal to 12.05. 
On purely technical grounds, several steps were required for a solution, making it difficult, 
as well as the presence of fractions/decimals in the resultant equation. However, judging by 
the number of blank responses I witnessed for this part, it appears that again, at the strategy 
stage of thinking, students encountered difficulty. Nevertheless, could there have been 
another factor, unrelated to the mathematics or thinking skills, making this a challenging 
task? It could be argued that the combined length of the perimeter and the platform-pole 
distance is not a useful measurement, therefore, being a value lacking meaningful applica¬ 
tion, part d) becomes intangible and strange. To highlight this view, it could be claimed that 
the perimeter of the platform’s cross-section is, by contrast, a meaningful measurement. 

(We may want to create some edging along this side of the platform, for instance.) However, 
without a connection to a justifiable purpose, perhaps the value required in part (d) was 
just too abstract? 

And finally, part (e) was difficult because it required students to identify the gradient 
of the pole from the equation, apply the concept of gradient as the ratio of rise to run and 
appeal to information already obtained in earlier parts of the question. Devising a strategy 
was indeed a challenge. However, the wording of part (e) may also have been problematic. 

Is “the length of the platform directly underneath it” clear? I think in hindsight, by introduc¬ 
ing a few labelled points in the diagram, and referencing them in the text of the question, 

I could have eliminated potential ambiguity. 

What made this question so difficult? Was it the unusual blend of familiar skills required 
for a solution? Was it the lack of experience with Cartesian ‘representation’? Or was it the 
manner of questioning itself? It is clear that we need our students to become better versed in 
problem solving strategies, as well as continue to develop their technical mastery, but we also 
need to ensure that the challenges are what we intend them to be, and not something else. 
This is where more analysis would help. 
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Conclusion 


What is required is dedicated research. I intend to create and test a range of extended 
response problems, from Year 9 through to Year 12, with their thinking skills, as well as their 
technical skills, fully unpacked. Student feedback on these problems, as well as cross-testing 
with other like problems, should help inform the accuracy of the skill set they purport to 
test. Indeed, in cases where students are misled by the posing of the question itself, criteria 
can hopefully be established for eliminating these distractions. The problems created, once 
clarified and well-directed, need to be unit based, as well as inter-unit based. Student 
performances on them could be compiled and analysed, together with information on past 
exposure to like problems. The results of such research would lead to better assessments 
providing truer indicators of ability. It would also hopefully lead to better classroom practices. 
A well-defined conceptual framework for creating problem solving tasks in a chosen area(s) 
of the syllabus would enable teachers to appropriately scaffold students through problem 
solving lessons. Moreover, by offering these sessions regularly, with problems accessible, 
but challenging, at a range of depths, students would hopefully find engagement in thinking, 
rather than merely attaining a mastery of technical routines at the next curriculum level. 

The end game has to be producing better skilled and more resilient thinkers of the future. 
That has to be a worthwhile pursuit. 


Appendix 

As a final case in point, consider this Mathematics Methods Unit 3-4 problem from the 2009 
trial paper produced by Insight. Several students had trouble with the question part shown, 
suggesting reasonable difficulty, yet the solution consisted of one simple line of arithmetic. 


The diagrams below show the beach with two markers. A and B at the start and end of 
the boat ramp, where the base of marker A is 6.5 metres further up the beach than the 
base of marker B. The line AB is perpendicular to the water’s edge. 



Keith records the horizontal distance of the water’s edge form the too marker A. 

He calculates that on a particular day the distance D metres of the water’s edge from 
the marker A is a function of time t (in minutes from when he starts to observe the 
waves). [Note: A function of the form D(t) = acos(t) + c relates to parts a) to d) of the 
question before the information below is introduced.] 

Due to winds, tides and currents, on some days the waves come further up the beach 
and are closer together. Keith observes that on such a day the distance of the water’s 
edge from marker A can be described by the equation: 

D = (8 + S) cosf^ 1 t) +11 

where S metres is the seasonal tidal factor which varies with the factors described above. 
S is normally distributed with a mean of 2 and a standard deviation of 0.3. 

On a particular day the waves just reach the top of the boat ramp at marker A. 

Find the value of S on this day. 

Figure 2: Sample question from the 2009 trial paper for Mathematics Methods Unit 3-4. 
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This question had lots of variables, with the above information split over two pages. 
In the end students had to realise that for the waves to just reach marker A , D(t) must 
have a minimum value of zero. 

PISA would classify the considerable thinking skills behind this as ‘communication’, 
for it involves deciphering a challenging amount of information spread over two pages, 
and ‘representation’. The mathematics at that point is almost trivial. D has a minimum 
of zero when the amplitude of the function is equal to the (positive) mean (i.e., when 
8+ S = ll). 

The difficulty level of this problem is not reducible to a competent grasp of arith¬ 
metic. Nor is it reducible to a sound knowledge of sinusoidal functions. It is thinking 
skills, and clearly we need to methodically factor this in if ‘item demand’ is to be 
properly evaluated. 
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